{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 图像分类\n",
    "\n",
    "在此项目中，你将对 [CIFAR-10 数据集](https://www.cs.toronto.edu/~kriz/cifar.html) 中的图片进行分类。该数据集包含飞机、猫狗和其他物体。你需要预处理这些图片，然后用所有样本训练一个卷积神经网络。图片需要标准化（normalized），标签需要采用 one-hot 编码。你需要应用所学的知识构建卷积的、最大池化（max pooling）、丢弃（dropout）和完全连接（fully connected）的层。最后，你需要在样本图片上看到神经网络的预测结果。\n",
    "\n",
    "\n",
    "## 获取数据\n",
    "\n",
    "请运行以下单元，以下载 [CIFAR-10 数据集（Python版）](https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz)。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All files found!\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "from urllib.request import urlretrieve\n",
    "from os.path import isfile, isdir\n",
    "from tqdm import tqdm\n",
    "import problem_unittests as tests\n",
    "import tarfile\n",
    "\n",
    "cifar10_dataset_folder_path = 'cifar-10-batches-py'\n",
    "\n",
    "# Use Floyd's cifar-10 dataset if present\n",
    "floyd_cifar10_location = '/input/cifar-10/python.tar.gz'\n",
    "if isfile(floyd_cifar10_location):\n",
    "    tar_gz_path = floyd_cifar10_location\n",
    "else:\n",
    "    tar_gz_path = 'cifar-10-python.tar.gz'\n",
    "\n",
    "class DLProgress(tqdm):\n",
    "    last_block = 0\n",
    "\n",
    "    def hook(self, block_num=1, block_size=1, total_size=None):\n",
    "        self.total = total_size\n",
    "        self.update((block_num - self.last_block) * block_size)\n",
    "        self.last_block = block_num\n",
    "\n",
    "if not isfile(tar_gz_path):\n",
    "    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:\n",
    "        urlretrieve(\n",
    "            'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',\n",
    "            tar_gz_path,\n",
    "            pbar.hook)\n",
    "\n",
    "if not isdir(cifar10_dataset_folder_path):\n",
    "    with tarfile.open(tar_gz_path) as tar:\n",
    "        tar.extractall()\n",
    "        tar.close()\n",
    "\n",
    "\n",
    "tests.test_folder_path(cifar10_dataset_folder_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 探索数据\n",
    "\n",
    "该数据集分成了几部分／批次（batches），以免你的机器在计算时内存不足。CIFAR-10 数据集包含 5 个部分，名称分别为 `data_batch_1`、`data_batch_2`，以此类推。每个部分都包含以下某个类别的标签和图片：\n",
    "\n",
    "* 飞机\n",
    "* 汽车\n",
    "* 鸟类\n",
    "* 猫\n",
    "* 鹿\n",
    "* 狗\n",
    "* 青蛙\n",
    "* 马\n",
    "* 船只\n",
    "* 卡车\n",
    "\n",
    "了解数据集也是对数据进行预测的必经步骤。你可以通过更改 `batch_id` 和 `sample_id` 探索下面的代码单元。`batch_id` 是数据集一个部分的 ID（1 到 5）。`sample_id` 是该部分中图片和标签对（label pair）的 ID。\n",
    "\n",
    "问问你自己：“可能的标签有哪些？”、“图片数据的值范围是多少？”、“标签是按顺序排列，还是随机排列的？”。思考类似的问题，有助于你预处理数据，并使预测结果更准确。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stats of batch 1:\n",
      "Samples: 10000\n",
      "Label Counts: {0: 1005, 1: 974, 2: 1032, 3: 1016, 4: 999, 5: 937, 6: 1030, 7: 1001, 8: 1025, 9: 981}\n",
      "First 20 Labels: [6, 9, 9, 4, 1, 1, 2, 7, 8, 3, 4, 7, 7, 2, 9, 9, 9, 3, 2, 6]\n",
      "\n",
      "Example of Image 5:\n",
      "Image - Min Value: 0 Max Value: 252\n",
      "Image - Shape: (32, 32, 3)\n",
      "Label - Label Id: 1 Name: automobile\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfoAAAH0CAYAAADVH+85AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHF9JREFUeJzt3UmPZOl1HuAvxsyMrKzKqsqau6rYA5vNbropkjJJmYIsUIBXWtn+BV7YO/8Yr73wymtDNAwIggwSMEmBNMeW2Wz2VOzumquyco6M2QttzI2Bc5gChYPn2Z88Ed+9cd+8q7ezWq0aAFBT9w/9AQCAfzyCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bh/T/0B/jH8l/+w79fZebGx9PwTK+f+3+pc/tGeGZvtJHa9faFYWruk1/+LDzznR/+PLVrbzILz/R6ybPvdFJzg7X18MylKzupXec34t/t83eupHb9+be+Hp6Zz+LXq7XWnu0fpeYGWxfDM+9+8NvUrr/97g/jQ8nnwNogN3dhMAjPDPuL1K5p4lrPZ7nfWFstU2NrvbXwzMkq/rxvrbUXp/F46eZ+Lu073/+75EH+P7t/3z8AAPzTJegBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGFl2+te3P84NddfxJuTBv1UUV67v5qEZ94f5yqQ3v7iK6m55TT+Ga/t5NraNlLfLXf22fa6k0n8PPZ3X6R2HXXiTWOT03Fq15e/+o3wzOzkNLXr2fPceVxbjzc3LqcHqV0ba/H7atlyrWtXt86l5r70ymvhmadP7qd2jceH4Zmjo1xLYevGW/laa22tPw/P3Lx+IbVrNrwanvngV/dSu86CN3oAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUFjZUpuPT9dScyfj/fDMsJMr92iLeKFCtzNMrXr228epuZ88+Cw88+snudKS1SReSpEtp1lfX0/NzebxopnWzf0/vb4Rv4f3xrlilR+983545sblXCHIZJ67ZpkCo7XkE24wSHzG3NG3L7z6amruc3fuhme2t0apXY8e3gvPLGe55+K5izdSc4tBvPRotJYr3rm5Ey8i+rSXO/uz4I0eAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgsLLtdeNeriFrtxtvJ+ssJqldl/vx4z93/mJq1+lxvJWvtdb2DuPf7eB0ltq1Spz9YpFok2ut9ZKfsZ/533gWb11rrbXjafzsz61yu370i1+GZ15/7bXUrjdevZOa6w/j7V+f+1yuGe54OQjPPH74NLXr4HCcmmvrm+GRP/6zt1Orfv7j74VnxvN4G2VrrR3Oci1vz4/jz8ZL41zD3q3eYXjm9Cjb2vj780YPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAorW2qz1tlNzd0YxYsYtlu8AKO11i5d3AjPfLyKlym01trmxjI1t9aJl6SMOrnbara5Fp+Z58ppTie5IqJF4n/jjVGupGO4Fr+vrt++kdp186Xb4ZlnR7lCkEcHuRKXb3zj6+GZ3cePUrv+9b/5Vnjmf/z3v07t+uEP/i41d+dLXw3PfPvtr6V2fXj/o/DMx9//cWrX/nQrNXc0jz/jvvjP42fYWmvj2YvwzM7OemrXWfBGDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUFjZ9rrhZu6rvbJ1NTzz8iq368Iw0Wa0/1lq12g73gzXWmvHw5PwzHKwSO364z+KN0lduxq/Xq219tEHH6TmPv3kfnim28u1G67m8Xa49W7u7P/kG/Gzfxq/NVprrf3oe99Nzb333p3wzGKc/JCbF8Mje8e5RsSjWe5964OHz8Mzx8teatfxPP4Zn+zlzmOyfi419/m7r4Rntq/dTO16+jx+9t/+9lupXWfBGz0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0BhZdvrjqa5xrALvc3wzOzZi9SuT/fiTWh/+uU3UrvG0+PU3K1lfGZ9tErt+uZ2/OzfvLKT2nWyzH3GZ2vxFsCT/dz9sZjGZ/rTw9Suu598HJ7Z2Jundl26sp2am/39z8Iz2ebAH/7q3fDMew8epHadznMtb/c/iTdZPnn+NLXr61/5Znjm7vbt1K7/9F//W2puOn4UnvnJj5+ldj1+/GF45qt/kXt2nwVv9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgsLKlNld666m5W60Xnjl/fiu16+cv4qUULyb7qV13r99Izf3bJy+HZwYHuQKdy+/Hz2Ptw4epXYvlLDX3uU58ZrBIDLXWuv34Pbzo5EpcJj/6aXjmQrKMZbkTLy9qrbXFPNGwdLBI7TrfOxeemRzn7vtL8UdOa6210Wocnjl49NvUrltffD08s7WZewZ//dVbqbkn+/EWqEdHJ6ldJye74ZmP3n8/tesseKMHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAorGx73Rtbo9Tc5vNn4ZleN9Gq1Vp7/aWXwjOHj5+mdrVVrkHtVmcVnhkNc7t6iUaozjL++VprLd5z9Q8m3cT/xsO11K7BKv7d+pmGt9baoBtv85tt5WrXVie51rv5JH4ei5a7F69143fItzdyrXzTzjA1t7h5LTyzfu9eatdJ5iMmWz3feuO11NyNk/g1uzGbp3a9/urN8MxrO/FGxLPijR4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFFa21Gb3wUepuck8XoIx7uWKRE4uxEsONk7i5SOttXb67oepuUVvEZ6Zb+Zuq24vXkqxlixx6bT11Nw8UQ60WOY+42owiM+kNuXm+ldfSe3a2su9X5wmLtn07sXUrovzo/DM5mmuKmm+lytWOXqyH545efD91K6H//sX4Znzb72e2vX8Ua64azq6FJ6Zj1Or2snzF+GZg0G2Suv3540eAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgsLLtdc+P9lJznx6fhmfmy1z71LBzPTwzuriT2vV8fJiau95bC89snOb+f1wcxJv5JtNcm1/byZ3j5uuvhWdOE01orbV29OwgPLO2jLfrtdZabzIJz0ye5u6ptpZrlOtsx9se+51cn9/yIP4c2Hgr1+bXhvHv1Vproyfx6rXj+/dTu/Z+/UF4ZvnJ49SurUtbqbnd7XhL5PNHud/mwyefhWdeHt5I7ToL3ugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKK9te9+I03j7VWmuPTuJtRrOD49SunWtXwjOr21dTu9Yu5hqh1g7izXz9B09Tu6ZHJ+GZoxZvrGqttcW5jdTc4O6d8Ey/s0jt2tyOn8fsN5+kds0SLYCn3Vxz4NafvZmaO9l7Fh9679epXW2eeAd6mPh8rbXJMte0Obh+Mzxz/V9+M7VrbaMXntn9zYepXdsn8V2ttXbhbrxp85NHuYa9jV68FXEwGKZ2nQVv9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgsLKlNrdvv5Sa6358PzyzMU6taotpvBhhrTNI7XpxfJCa+8Gnn4Vnbp4epna90eIHOUmUsbTW2vh+/Dq31tr0p7+K72rx69xaa51bt8Izp69fT+06mY/CM2+/miunOe6eS82NH9wLzwz3c+VW8/PxApLpJ8lCoce5UqzB1SfhmZNruVKswaUL4ZmLf/HV1K69Tx+m5rZ34mU4Xz13N7Xrb/7Xi/DM2na8xOyseKMHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAorGx73fWb11Jzh/efhWdGFzupXa2zFh4ZdHO7Hj57npr7z7/4P+GZL1zOtZP9x/XN8Mwo+a/q6vgoNbf7Try9bvdKvPmrtdY+msRbzabJprybr98Mz9y5mPte04ePU3PnEq1mneU0tasdxn9na92N1KqD8UlqbvHRR+GZ1YNHqV0vtuLPqs0v5BpEb778amru9FH8vroyij9zWmvtK196LTxz++XceZwFb/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoLCypTb7ixepuf5qPzwz6OeOcdqLF5DszcepXbvjXNnJfBX/bgeDXLnH/cEoPLO9mqd2Tbu5udVqEp7ZX+ZKSz57Ei+1Od9dT+16kbhkf3X/r1K7vnDrVmru1Uvx73Z57Xpq1/G9++GZxTh+vVprbbXI3YsvXjxN7Mo9B6br8VKb2X68IKy11qa/fD81N0oUOk3WB6ldd998Kzwze/Db1K6z4I0eAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgsLLtdcPVMjXXX87CMzvdXAPStBdvrerPpqldJ6e587h15Up45qWXb6d23T9KNPOtcm1cw2RrVWce/8lMl/HGu9Zau3F5JzzTzxWhtYOnj8Izq91cK9+D57mWt/3RMDxzZxL/PbfWWvdZvL2ujXOH353n3rfG8/g5nixyz49VohVxNO6kdj28/1lqbtSJ7zue567Z9iQ+t/P266ldZ8EbPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAorGypzcZ4lJp7ML8QnrnaPU3tujjeC8/0nzxM7ZofvkjNffHNl8Mzd77w+dSu3V+8F5650emldrVBrgxnsIr/b7xxlCtx6bf4ZxyNNlK7fvPhvfDMznHuPeGVz11KzX02jBfUPP4g93vZONwNz3TmuXuqs8jdw6eJUqxpN3fNpsfxXbuLw9Su0eh8au5wGi+POp7krtnu/cfhmf6d66ldZ8EbPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGFl2+v2j+NNV6219t39eEvT/HJqVfvWchqe2XjyKLVrfXaSmvvK174dnrl5+7XUru/86J3wzP4k1xy46Ofuj1miLW9j1UntOv0sfq17l3LNcK9c3AnPnC72U7v6m8PU3Nt/+vXwzG680Owf5n7yJDwzWeaa0Jb9tdTcOHFfbW4mH1Ybm+GR8TDXyre8fDE1d9ri+x49jbcUttba/t6z8MyLX7+f2vWXqanf5Y0eAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgsLLtddODB6m5D54/Ds+MZ7k2ru2X4o1hXx7kWte2+vFWvtZae/n27fDM+XO5BrXJIt7mNzmJz7TW2nCwSM2druL7ht3c/TGcxq/ZeDfXxtXtxx8Fy16ure3x81wD44t3fxWeGa3nGtQO18/FZzZGqV2Tc1upuePj4/DMaCf329ydxlsiD+e531h3Nk7NPXx0FN+1Hm/la621g1n8ObB5kGt7PAve6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYWVLbf7V3VxZwdPdeJnFjz8+Se36m3vxkoONV3Lfa3RuLTW31YsXdcwO4wUYrbW26MRLMI4nuV3rvdytv+gl/jfu5P6fXnbjc7vH8WKP1lpbncYLdIbHubOf7eWKiFYffhKeGSXfZaaj8+GZd+aT1K57z56k5taX8ZnhMlcYM1iP/146s05q1+lerpjpeBUvB+qfG6R2LQbx73b34nZq11nwRg8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFBY2fa612/mvtq/G90Jz9xeu5/a9T/fizeN/e29WWrXH929mZo7+vDj8Mxe8v/H3jJex7U3zTUHXhnFm65aa22x6oVnZsvcNXu6ip/Hs1G8fbG11k778fa6rU7uN7Z5IXf2y2n8M7bnB6lda2vxlsjPTnPNcM8Xq9Tc9UG8eW20mbs/tjbj57Ea59oNn01z59jvxZ8Fvd3c8+NLq2F45txh7jlwFrzRA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCypbaTJJlJ5fWO+GZP3l9J7Xr2XG8tOQn9/dTu959/CI19/lEUcd0mLutVsv4/52Hp5Pcrkm8lKK11gbr8e+2WuZKS1pibmNtPbXqcBUvIDm4cy216/Jbb6TmevGfS3vnr7+X2nU7cV+9dPFKalebTFNj6/34gezPcoUxx8/jz9PryYKlmzuXU3PDbvy3OdjNPU/vHsYLyW5vb6d2nQVv9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIWVba/r9HJfrTOPt1bd2M41hv2Lly+EZw6m8Zax1lq7t5dr8zvpxdv8rt6+ndrVG47CM6fzXDPc6eFhaq4/W4RnhoON1K743dHa/PHT1K7zi3l4ZnKQu6d2Z4kautba9sWL8ZlO7l1mcBr/brc2N1O7hsn3rc7mWnxmkPuM3aN4w961fvz33FpriQLR1lpr3Un8t3mSfA5c6MXvj1fv5HLiLHijB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKKxse91qlatAWi0T7WTLeONda629eSl+/E9vnEvtOp7kPuN8HG/L27l8JbVr/Vy8r21vmWuvm01nqbl5Ym7SyzUOdju98Mz55L/umV6t6cF+btlp7jxWj56EZ15quefAoBdv89sa587jai/Xbvgi0Ui5thVvAGytteUsfmPNT/ZSuw4muVbERHldW06OU7tuvHk1PPPyndxz8Sx4oweAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhZUttVl2cv/DLFq8SKTNcwUpF/rxwo2v3N5J7Xp+uJuamz5+GJ6ZHeeKIoab8XKP0+R1nq1yc91l/FovZom2jdZaZxG/P+bJ85gOMuUv8eKX1lrrzHPnsegN40PdXKnNYh7/bqtkWc/6YpCaW82m4ZlH67mimdla/OyXa6lVbbCZO4+Tk/h5DFfL1K4rd66HZ9b7ifv3jHijB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKKxse91wYzM111sfhWeme0epXZlWs5vb8c/XWmv/bD/XrPXu3uPwzKMHn6R2HYwPwjNHy1z71Gk39z/uYLkKz8xXuba27ir+8zzu5NraTlbxuX7yPWE5yV2z5SR+D3eS7XUtcZ1P+7nrvEw05bXW2nHmM65NUrtaN/7d1ge5+rrlIt5C11prm8v4d3vt2lZq18Vh/OxPnueaA3Of8Hd5oweAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhZUttWndXmqs0xmEZ/obqVXttDsLzwwSZQqttXbnRq4M5+PP4gUT08lxatdiGd+1N88VYDzr5G79rV78vuqscteskyio2c/1xbRH03hpSbeTe0/oJQp0srJvMoMWv86Pl/Hfc2ut7bdcGc5R4lrfSpb8bCcKuHq7h6ld1/rrqbmv3b4ennn1du7hPRrHi8wmybIepTYAwP+XoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhdVtr1vm/oeZjE/CM9k2rk6iSWo1zTVkndvcTM3tnI83Lu0+fZLadfgoPrffy13nHySbxi4miujOJxoRW2ttM9FeN+vmmvIO5vG502TrWra7rteNX+thom2wtdZGqU+Z29Xv5CoHR4lrvZzNU7umi/h5bCTvjwvncp+xzQ7CI0cvcmd/cD7+m+7Mc8+cndTU7/JGDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUFjZ9rrFMtfitUrMdZINasP+MDyzGucakFruONrVzfhn/Ok7f5/a9fzB0/DMvJO7hZ8mO9QO5vE2v9Ei2U6W+IhryXtxNYxf526iTa611jqJVr7WWuv3441hi1WynWwR/53N57m2tlXyMw4zx59sr1sm7qtuP/fQWbbcM27vaC8801vlzmOtuxWe6Sz/cHHrjR4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFFa21KY7iBdgtNbaINHD0EkWxnR6ieNf5IozFsdHqbkbW6PwzOVB7jMOTsfhmfPLXEHKaSf3P243MTfv50pLjpfxuXHyXmyJEpfePLeskywU6iYKhVarZLlVJ372uW/V2qDTy80lnh8byfv+XGJss5N8DuTGWmvxwcn4OLUp8zgddePP0rPijR4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaCwuu11/dxX660S//uscu1kLdVel2vl63dz3VrnOvHGsD9762Zq1/5JfNfPPnmW2vVsMk/NnS7jbWiTZK/ZMnF/LJP/uy8S36ubrG3sJGveut1sNV9cL9Hy1k9+vI1u7lk16safBVv93OFvdePPuMvJdBklb5BBi/+mh8l7arWI7zpNtHOeFW/0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaCwsqU2bbieHIyXFXRWyTaLRPHOfD5LrVomL3WmvOHGKLWq/eWXb4Vnrg1yhUIfPD5IzT0+jp//i3mupON02QvPTJK34rwTv86rRPFLa611e/Hv1VprvcRcsj+nDRIlP/1kt9VmptyqtbaWOP+1Tu5Dnu8twjMXkwU6m73cfbU+iJ9jP3crttks/hw46cTP8Kx4oweAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6ACiss8o2rwEA/+R5oweAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bh/xfkBwlHN40TWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f02183bf240>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 253
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import helper\n",
    "import numpy as np\n",
    "\n",
    "# Explore the dataset\n",
    "batch_id = 1\n",
    "sample_id = 5\n",
    "helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实现预处理函数\n",
    "\n",
    "### 标准化\n",
    "\n",
    "在下面的单元中，实现 `normalize` 函数，传入图片数据 `x`，并返回标准化 Numpy 数组。值应该在 0 到 1 的范围内（含 0 和 1）。返回对象应该和 `x` 的形状一样。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "def normalize(x):\n",
    "    \"\"\"\n",
    "    Normalize a list of sample image data in the range of 0 to 1\n",
    "    : x: List of image data.  The image shape is (32, 32, 3)\n",
    "    : return: Numpy array of normalize data\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    img_raw = np.array(x)\n",
    "    pixel_max = img_raw.max()\n",
    "    pixel_min = img_raw.min()\n",
    "    img_normalized = (img_raw - pixel_min) / (pixel_max - pixel_min)\n",
    "    #print(img_normalized)\n",
    "    return img_normalized\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_normalize(normalize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-hot 编码\n",
    "\n",
    "和之前的代码单元一样，你将为预处理实现一个函数。这次，你将实现 `one_hot_encode` 函数。输入，也就是 `x`，是一个标签列表。实现该函数，以返回为 one_hot 编码的 Numpy 数组的标签列表。标签的可能值为 0 到 9。每次调用 `one_hot_encode` 时，对于每个值，one_hot 编码函数应该返回相同的编码。确保将编码映射保存到该函数外面。\n",
    "\n",
    "提示：不要重复发明轮子。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import label_binarize\n",
    "# http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.label_binarize.html#sklearn.preprocessing.label_binarize\n",
    "label_one_hot_classes = range(10)\n",
    "def one_hot_encode(x):\n",
    "    \"\"\"\n",
    "    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.\n",
    "    : x: List of sample Labels\n",
    "    : return: Numpy array of one-hot encoded labels\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    # print(label_binarize(x, classes = label_one_hot_classes))\n",
    "    return label_binarize(x, classes = label_one_hot_classes)\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_one_hot_encode(one_hot_encode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 随机化数据\n",
    "\n",
    "之前探索数据时，你已经了解到，样本的顺序是随机的。再随机化一次也不会有什么关系，但是对于这个数据集没有必要。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预处理所有数据并保存\n",
    "\n",
    "运行下方的代码单元，将预处理所有 CIFAR-10 数据，并保存到文件中。下面的代码还使用了 10% 的训练数据，用来验证。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "# Preprocess Training, Validation, and Testing Data\n",
    "helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 检查点\n",
    "\n",
    "这是你的第一个检查点。如果你什么时候决定再回到该记事本，或需要重新启动该记事本，你可以从这里开始。预处理的数据已保存到本地。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import pickle\n",
    "import problem_unittests as tests\n",
    "import helper\n",
    "\n",
    "# Load the Preprocessed Validation data\n",
    "valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建网络\n",
    "\n",
    "对于该神经网络，你需要将每层都构建为一个函数。你看到的大部分代码都位于函数外面。要更全面地测试你的代码，我们需要你将每层放入一个函数中。这样使我们能够提供更好的反馈，并使用我们的统一测试检测简单的错误，然后再提交项目。\n",
    "\n",
    ">**注意**：如果你觉得每周很难抽出足够的时间学习这门课程，我们为此项目提供了一个小捷径。对于接下来的几个问题，你可以使用 [TensorFlow Layers](https://www.tensorflow.org/api_docs/python/tf/layers) 或 [TensorFlow Layers (contrib)](https://www.tensorflow.org/api_guides/python/contrib.layers) 程序包中的类来构建每个层级，但是“卷积和最大池化层级”部分的层级除外。TF Layers 和 Keras 及 TFLearn 层级类似，因此很容易学会。\n",
    "\n",
    ">但是，如果你想充分利用这门课程，请尝试自己解决所有问题，不使用 TF Layers 程序包中的任何类。你依然可以使用其他程序包中的类，这些类和你在 TF Layers 中的类名称是一样的！例如，你可以使用 TF Neural Network 版本的 `conv2d` 类 [tf.nn.conv2d](https://www.tensorflow.org/api_docs/python/tf/nn/conv2d)，而不是 TF Layers 版本的 `conv2d` 类 [tf.layers.conv2d](https://www.tensorflow.org/api_docs/python/tf/layers/conv2d)。\n",
    "\n",
    "我们开始吧！\n",
    "\n",
    "\n",
    "### 输入\n",
    "\n",
    "神经网络需要读取图片数据、one-hot 编码标签和丢弃保留概率（dropout keep probability）。请实现以下函数：\n",
    "\n",
    "* 实现 `neural_net_image_input`\n",
    " * 返回 [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder)\n",
    " * 使用 `image_shape` 设置形状，部分大小设为 `None`\n",
    " * 使用 [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder) 中的 TensorFlow `name` 参数对 TensorFlow 占位符 \"x\" 命名\n",
    "* 实现 `neural_net_label_input`\n",
    " * 返回 [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder)\n",
    " * 使用 `n_classes` 设置形状，部分大小设为 `None`\n",
    " * 使用 [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder) 中的 TensorFlow `name` 参数对 TensorFlow 占位符 \"y\" 命名\n",
    "* 实现 `neural_net_keep_prob_input`\n",
    " * 返回 [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder)，用于丢弃保留概率\n",
    " * 使用 [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder) 中的 TensorFlow `name` 参数对 TensorFlow 占位符 \"keep_prob\" 命名\n",
    "\n",
    "这些名称将在项目结束时，用于加载保存的模型。\n",
    "\n",
    "注意：TensorFlow 中的 `None` 表示形状可以是动态大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image Input Tests Passed.\n",
      "Label Input Tests Passed.\n",
      "Keep Prob Tests Passed.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "def neural_net_image_input(image_shape):\n",
    "    \"\"\"\n",
    "    Return a Tensor for a batch of image input\n",
    "    : image_shape: Shape of the images\n",
    "    : return: Tensor for image input.\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "\n",
    "    return tf.placeholder(tf.float32, shape = [None, *image_shape], name = \"x\")\n",
    "\n",
    "\n",
    "def neural_net_label_input(n_classes):\n",
    "    \"\"\"\n",
    "    Return a Tensor for a batch of label input\n",
    "    : n_classes: Number of classes\n",
    "    : return: Tensor for label input.\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return tf.placeholder(tf.float32, shape = [None, n_classes], name = \"y\")\n",
    "\n",
    "\n",
    "def neural_net_keep_prob_input():\n",
    "    \"\"\"\n",
    "    Return a Tensor for keep probability\n",
    "    : return: Tensor for keep probability.\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return tf.placeholder(tf.float32, shape = None, name = \"keep_prob\")\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tf.reset_default_graph()\n",
    "tests.test_nn_image_inputs(neural_net_image_input)\n",
    "tests.test_nn_label_inputs(neural_net_label_input)\n",
    "tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 卷积和最大池化层\n",
    "\n",
    "卷积层级适合处理图片。对于此代码单元，你应该实现函数 `conv2d_maxpool` 以便应用卷积然后进行最大池化：\n",
    "\n",
    "* 使用 `conv_ksize`、`conv_num_outputs` 和 `x_tensor` 的形状创建权重（weight）和偏置（bias）。\n",
    "* 使用权重和 `conv_strides` 对 `x_tensor` 应用卷积。\n",
    " * 建议使用我们建议的间距（padding），当然也可以使用任何其他间距。\n",
    "* 添加偏置\n",
    "* 向卷积中添加非线性激活（nonlinear activation）\n",
    "* 使用 `pool_ksize` 和 `pool_strides` 应用最大池化\n",
    " * 建议使用我们建议的间距（padding），当然也可以使用任何其他间距。\n",
    "\n",
    "**注意**：对于**此层**，**请勿使用** [TensorFlow Layers](https://www.tensorflow.org/api_docs/python/tf/layers) 或 [TensorFlow Layers (contrib)](https://www.tensorflow.org/api_guides/python/contrib.layers)，但是仍然可以使用 TensorFlow 的 [Neural Network](https://www.tensorflow.org/api_docs/python/tf/nn) 包。对于所有**其他层**，你依然可以使用快捷方法。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):\n",
    "    \"\"\"\n",
    "    Apply convolution then max pooling to x_tensor\n",
    "    :param x_tensor: TensorFlow Tensor\n",
    "    :param conv_num_outputs: Number of outputs for the convolutional layer\n",
    "    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer\n",
    "    :param conv_strides: Stride 2-D Tuple for convolution\n",
    "    :param pool_ksize: kernal size 2-D Tuple for pool\n",
    "    :param pool_strides: Stride 2-D Tuple for pool\n",
    "    : return: A tensor that represents convolution and max pooling of x_tensor\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    W_conv = tf.Variable(tf.truncated_normal(shape = [*conv_ksize,int(x_tensor.shape[3]), conv_num_outputs]))\n",
    "    b_conv = tf.Variable(tf.constant(0.1, shape = [conv_num_outputs]))\n",
    "    h_conv = tf.nn.relu(tf.nn.conv2d(x_tensor, W_conv, [1, *conv_strides, 1], padding = \"SAME\") + b_conv)\n",
    "    h_pool = tf.nn.max_pool(h_conv, [1, *pool_ksize, 1], [1, *pool_strides, 1], padding = \"SAME\")\n",
    "    return h_pool\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_con_pool(conv2d_maxpool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 扁平化层\n",
    "\n",
    "实现 `flatten` 函数，将 `x_tensor` 的维度从四维张量（4-D tensor）变成二维张量。输出应该是形状（*部分大小（Batch Size）*，*扁平化图片大小（Flattened Image Size）*）。快捷方法：对于此层，你可以使用 [TensorFlow Layers](https://www.tensorflow.org/api_docs/python/tf/layers) 或 [TensorFlow Layers (contrib)](https://www.tensorflow.org/api_guides/python/contrib.layers) 包中的类。如果你想要更大挑战，可以仅使用其他 TensorFlow 程序包。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def flatten(x_tensor):\n",
    "    \"\"\"\n",
    "    Flatten x_tensor to (Batch Size, Flattened Image Size)\n",
    "    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.\n",
    "    : return: A tensor of size (Batch Size, Flattened Image Size).\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    flattened_image_size = int(x_tensor.shape[1] \n",
    "                               * x_tensor.shape[2]\n",
    "                               * x_tensor.shape[3])\n",
    "    return (tf.reshape(x_tensor, shape = [-1, flattened_image_size]))\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_flatten(flatten)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 全连接层\n",
    "\n",
    "实现 `fully_conn` 函数，以向 `x_tensor` 应用完全连接的层级，形状为（*部分大小（Batch Size）*，*num_outputs*）。快捷方法：对于此层，你可以使用 [TensorFlow Layers](https://www.tensorflow.org/api_docs/python/tf/layers) 或 [TensorFlow Layers (contrib)](https://www.tensorflow.org/api_guides/python/contrib.layers) 包中的类。如果你想要更大挑战，可以仅使用其他 TensorFlow 程序包。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def fully_conn(x_tensor, num_outputs):\n",
    "    \"\"\"\n",
    "    Apply a fully connected layer to x_tensor using weight and bias\n",
    "    : x_tensor: A 2-D tensor where the first dimension is batch size.\n",
    "    : num_outputs: The number of output that the new tensor should be.\n",
    "    : return: A 2-D tensor where the second dimension is num_outputs.\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    W_fc = tf.Variable(\n",
    "        tf.truncated_normal(\n",
    "            shape = [int(x_tensor.shape[1]), num_outputs]))\n",
    "    b_fc = tf.Variable(tf.constant(0.1, shape = [num_outputs]))\n",
    "    h_fc = tf.nn.relu(tf.matmul(x_tensor, W_fc) + b_fc)\n",
    "    return h_fc\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_fully_conn(fully_conn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输出层\n",
    "\n",
    "实现 `output` 函数，向 x_tensor 应用完全连接的层级，形状为（*部分大小（Batch Size）*，*num_outputs*）。快捷方法：对于此层，你可以使用 [TensorFlow Layers](https://www.tensorflow.org/api_docs/python/tf/layers) 或 [TensorFlow Layers (contrib)](https://www.tensorflow.org/api_guides/python/contrib.layers) 包中的类。如果你想要更大挑战，可以仅使用其他 TensorFlow 程序包。\n",
    "\n",
    "**注意**：该层级不应应用 Activation、softmax 或交叉熵（cross entropy）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def output(x_tensor, num_outputs):\n",
    "    \"\"\"\n",
    "    Apply a output layer to x_tensor using weight and bias\n",
    "    : x_tensor: A 2-D tensor where the first dimension is batch size.\n",
    "    : num_outputs: The number of output that the new tensor should be.\n",
    "    : return: A 2-D tensor where the second dimension is num_outputs.\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    return fully_conn(x_tensor, num_outputs)\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_output(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建卷积模型\n",
    "\n",
    "实现函数 `conv_net`， 创建卷积神经网络模型。该函数传入一批图片 `x`，并输出对数（logits）。使用你在上方创建的层创建此模型：\n",
    "\n",
    "* 应用 1、2 或 3 个卷积和最大池化层（Convolution and Max Pool layers）\n",
    "* 应用一个扁平层（Flatten Layer）\n",
    "* 应用 1、2 或 3 个完全连接层（Fully Connected Layers）\n",
    "* 应用一个输出层（Output Layer）\n",
    "* 返回输出\n",
    "* 使用 `keep_prob` 向模型中的一个或多个层应用 [TensorFlow 的 Dropout](https://www.tensorflow.org/api_docs/python/tf/nn/dropout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-82-62736fa6ec7b>:63: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See tf.nn.softmax_cross_entropy_with_logits_v2.\n",
      "\n",
      "Neural Network Built!\n"
     ]
    }
   ],
   "source": [
    "def conv_net(x, keep_prob):\n",
    "    \"\"\"\n",
    "    Create a convolutional neural network model\n",
    "    : x: Placeholder tensor that holds image data.\n",
    "    : keep_prob: Placeholder tensor that hold dropout keep probability.\n",
    "    : return: Tensor that represents logits\n",
    "    \"\"\"\n",
    "    # TODO: Apply 1, 2, or 3 Convolution and Max Pool layers\n",
    "    #    Play around with different number of outputs, kernel size and stride\n",
    "    # Function Definition from Above:\n",
    "    #    conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)\n",
    "    conv1 = conv2d_maxpool(x, 32, [3, 3], [1, 1], [2, 2], [1, 1])\n",
    "    \n",
    "\n",
    "    # TODO: Apply a Flatten Layer\n",
    "    # Function Definition from Above:\n",
    "    #   flatten(x_tensor)\n",
    "    flat1 = flatten(conv1)\n",
    "    \n",
    "\n",
    "    # TODO: Apply 1, 2, or 3 Fully Connected Layers\n",
    "    #    Play around with different number of outputs\n",
    "    # Function Definition from Above:\n",
    "    #   fully_conn(x_tensor, num_outputs)\n",
    "    fc1 = fully_conn(flat1, 10)\n",
    "    \n",
    "    fc1_drop = tf.nn.dropout(fc1, keep_prob)\n",
    "    \n",
    "    # TODO: Apply an Output Layer\n",
    "    #    Set this to the number of classes\n",
    "    # Function Definition from Above:\n",
    "    #   output(x_tensor, num_outputs)\n",
    "    y_out = output(fc1, 10)\n",
    "    \n",
    "    \n",
    "    # TODO: return output\n",
    "    return y_out\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "\n",
    "##############################\n",
    "## Build the Neural Network ##\n",
    "##############################\n",
    "\n",
    "# Remove previous weights, bias, inputs, etc..\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Inputs\n",
    "x = neural_net_image_input((32, 32, 3))\n",
    "y = neural_net_label_input(10)\n",
    "keep_prob = neural_net_keep_prob_input()\n",
    "\n",
    "# Model\n",
    "logits = conv_net(x, keep_prob)\n",
    "\n",
    "# Name logits Tensor, so that is can be loaded from disk after training\n",
    "logits = tf.identity(logits, name='logits')\n",
    "\n",
    "# Loss and Optimizer\n",
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))\n",
    "optimizer = tf.train.AdamOptimizer().minimize(cost)\n",
    "\n",
    "# Accuracy\n",
    "correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')\n",
    "\n",
    "tests.test_conv_net(conv_net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练神经网络\n",
    "\n",
    "### 单次优化\n",
    "\n",
    "实现函数 `train_neural_network` 以进行单次优化（single optimization）。该优化应该使用 `optimizer` 优化 `session`，其中 `feed_dict` 具有以下参数：\n",
    "\n",
    "* `x` 表示图片输入\n",
    "* `y` 表示标签\n",
    "* `keep_prob` 表示丢弃的保留率\n",
    "\n",
    "每个部分都会调用该函数，所以 `tf.global_variables_initializer()` 已经被调用。\n",
    "\n",
    "注意：不需要返回任何内容。该函数只是用来优化神经网络。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):\n",
    "    \"\"\"\n",
    "    Optimize the session on a batch of images and labels\n",
    "    : session: Current TensorFlow session\n",
    "    : optimizer: TensorFlow optimizer function\n",
    "    : keep_probability: keep probability\n",
    "    : feature_batch: Batch of Numpy image data\n",
    "    : label_batch: Batch of Numpy label data\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    session.run(optimizer, feed_dict = {x: feature_batch, y: label_batch, keep_prob: keep_probability})\n",
    "    pass\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_train_nn(train_neural_network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 显示数据\n",
    "\n",
    "实现函数 `print_stats` 以输出损失和验证准确率。使用全局变量 `valid_features` 和 `valid_labels` 计算验证准确率。使用保留率 `1.0` 计算损失和验证准确率（loss and validation accuracy）。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_stats(session, feature_batch, label_batch, cost, accuracy):\n",
    "    \"\"\"\n",
    "    Print information about loss and validation accuracy\n",
    "    : session: Current TensorFlow session\n",
    "    : feature_batch: Batch of Numpy image data\n",
    "    : label_batch: Batch of Numpy label data\n",
    "    : cost: TensorFlow cost function\n",
    "    : accuracy: TensorFlow accuracy function\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    print(\" loss: %g\"%session.run(cost, feed_dict = {x: feature_batch, y: label_batch, keep_prob:1.0}))\n",
    "    print(\" accuracy: %g\"%session.run(accuracy, feed_dict = {x: valid_features, y: valid_labels, keep_prob: 1.0}))\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 超参数\n",
    "\n",
    "调试以下超参数：\n",
    "* 设置 `epochs` 表示神经网络停止学习或开始过拟合的迭代次数\n",
    "* 设置 `batch_size`，表示机器内存允许的部分最大体积。大部分人设为以下常见内存大小：\n",
    "\n",
    " * 64\n",
    " * 128\n",
    " * 256\n",
    " * ...\n",
    "* 设置 `keep_probability` 表示使用丢弃时保留节点的概率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Tune Parameters\n",
    "epochs = 1000\n",
    "batch_size = 32\n",
    "keep_probability = 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 在单个 CIFAR-10 部分上训练\n",
    "\n",
    "我们先用单个部分，而不是用所有的 CIFAR-10 批次训练神经网络。这样可以节省时间，并对模型进行迭代，以提高准确率。最终验证准确率达到 50% 或以上之后，在下一部分对所有数据运行模型。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the Training on a Single Batch...\n",
      "Epoch  1, CIFAR-10 Batch 1:  loss: 2.31032\n",
      "accuracy: 0.0998\n",
      "Epoch  2, CIFAR-10 Batch 1:  loss: 2.30999\n",
      "accuracy: 0.0996\n",
      "Epoch  3, CIFAR-10 Batch 1:  loss: 2.31029\n",
      "accuracy: 0.0996\n",
      "Epoch  4, CIFAR-10 Batch 1:  loss: 2.31068\n",
      "accuracy: 0.0996\n",
      "Epoch  5, CIFAR-10 Batch 1:  loss: 2.31081\n",
      "accuracy: 0.0996\n",
      "Epoch  6, CIFAR-10 Batch 1:  loss: 2.31086\n",
      "accuracy: 0.0996\n",
      "Epoch  7, CIFAR-10 Batch 1:  loss: 2.31085\n",
      "accuracy: 0.0996\n",
      "Epoch  8, CIFAR-10 Batch 1:  loss: 2.31078\n",
      "accuracy: 0.0996\n",
      "Epoch  9, CIFAR-10 Batch 1:  loss: 2.31075\n",
      "accuracy: 0.0996\n",
      "Epoch 10, CIFAR-10 Batch 1:  loss: 2.31068\n",
      "accuracy: 0.0996\n",
      "Epoch 11, CIFAR-10 Batch 1:  loss: 2.31061\n",
      "accuracy: 0.0996\n",
      "Epoch 12, CIFAR-10 Batch 1:  loss: 2.31055\n",
      "accuracy: 0.0996\n",
      "Epoch 13, CIFAR-10 Batch 1:  loss: 2.31048\n",
      "accuracy: 0.0996\n",
      "Epoch 14, CIFAR-10 Batch 1:  loss: 2.31043\n",
      "accuracy: 0.0996\n",
      "Epoch 15, CIFAR-10 Batch 1:  loss: 2.31038\n",
      "accuracy: 0.0996\n",
      "Epoch 16, CIFAR-10 Batch 1:  loss: 2.31034\n",
      "accuracy: 0.0996\n",
      "Epoch 17, CIFAR-10 Batch 1:  loss: 2.31031\n",
      "accuracy: 0.0996\n",
      "Epoch 18, CIFAR-10 Batch 1:  loss: 2.31028\n",
      "accuracy: 0.0996\n",
      "Epoch 19, CIFAR-10 Batch 1:  loss: 2.31026\n",
      "accuracy: 0.0996\n",
      "Epoch 20, CIFAR-10 Batch 1:  loss: 2.31024\n",
      "accuracy: 0.0996\n",
      "Epoch 21, CIFAR-10 Batch 1:  loss: 2.31023\n",
      "accuracy: 0.0996\n",
      "Epoch 22, CIFAR-10 Batch 1:  loss: 2.31021\n",
      "accuracy: 0.0998\n",
      "Epoch 23, CIFAR-10 Batch 1:  loss: 2.3102\n",
      "accuracy: 0.0998\n",
      "Epoch 24, CIFAR-10 Batch 1:  loss: 2.3102\n",
      "accuracy: 0.0998\n",
      "Epoch 25, CIFAR-10 Batch 1:  loss: 2.31019\n",
      "accuracy: 0.0998\n",
      "Epoch 26, CIFAR-10 Batch 1:  loss: 2.31018\n",
      "accuracy: 0.0998\n",
      "Epoch 27, CIFAR-10 Batch 1:  loss: 2.31018\n",
      "accuracy: 0.0998\n",
      "Epoch 28, CIFAR-10 Batch 1:  loss: 2.31018\n",
      "accuracy: 0.0998\n",
      "Epoch 29, CIFAR-10 Batch 1:  loss: 2.31017\n",
      "accuracy: 0.0998\n",
      "Epoch 30, CIFAR-10 Batch 1:  loss: 2.31017\n",
      "accuracy: 0.0998\n",
      "Epoch 31, CIFAR-10 Batch 1:  loss: 2.31017\n",
      "accuracy: 0.0998\n",
      "Epoch 32, CIFAR-10 Batch 1:  loss: 2.31017\n",
      "accuracy: 0.0998\n",
      "Epoch 33, CIFAR-10 Batch 1:  loss: 2.31017\n",
      "accuracy: 0.0998\n",
      "Epoch 34, CIFAR-10 Batch 1:  loss: 2.31017\n",
      "accuracy: 0.0998\n",
      "Epoch 35, CIFAR-10 Batch 1:  loss: 2.31016\n",
      "accuracy: 0.0998\n",
      "Epoch 36, CIFAR-10 Batch 1:  loss: 2.31016\n",
      "accuracy: 0.0998\n",
      "Epoch 37, CIFAR-10 Batch 1:  loss: 2.31016\n",
      "accuracy: 0.0998\n",
      "Epoch 38, CIFAR-10 Batch 1:  loss: 2.31016\n",
      "accuracy: 0.0998\n",
      "Epoch 39, CIFAR-10 Batch 1:  loss: 2.31016\n",
      "accuracy: 0.0998\n",
      "Epoch 40, CIFAR-10 Batch 1:  loss: 2.31016\n",
      "accuracy: 0.0998\n",
      "Epoch 41, CIFAR-10 Batch 1:  loss: 2.31016\n",
      "accuracy: 0.0996\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-103-6aa082bf118d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     11\u001b[0m         \u001b[0mbatch_i\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mbatch_features\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_labels\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mhelper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_preprocess_training_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_i\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m             \u001b[0mtrain_neural_network\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_probability\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_features\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_labels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Epoch {:>2}, CIFAR-10 Batch {}:  '\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepoch\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_i\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m''\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0mprint_stats\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_features\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_labels\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcost\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccuracy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-89-43584b18ee24>\u001b[0m in \u001b[0;36mtrain_neural_network\u001b[0;34m(session, optimizer, keep_probability, feature_batch, label_batch)\u001b[0m\n\u001b[1;32m      9\u001b[0m     \"\"\"\n\u001b[1;32m     10\u001b[0m     \u001b[0;31m# TODO: Implement Function\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m     \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mfeature_batch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mlabel_batch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_prob\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mkeep_probability\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m     \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "print('Checking the Training on a Single Batch...')\n",
    "with tf.Session() as sess:\n",
    "    # Initializing the variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Training cycle\n",
    "    for epoch in range(epochs):\n",
    "        batch_i = 1\n",
    "        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):\n",
    "            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)\n",
    "        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')\n",
    "        print_stats(sess, batch_features, batch_labels, cost, accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 完全训练模型\n",
    "\n",
    "现在，单个 CIFAR-10 部分的准确率已经不错了，试试所有五个部分吧。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "save_model_path = './image_classification'\n",
    "\n",
    "print('Training...')\n",
    "with tf.Session() as sess:\n",
    "    # Initializing the variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Training cycle\n",
    "    for epoch in range(epochs):\n",
    "        # Loop over all batches\n",
    "        n_batches = 5\n",
    "        for batch_i in range(1, n_batches + 1):\n",
    "            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):\n",
    "                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)\n",
    "            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')\n",
    "            print_stats(sess, batch_features, batch_labels, cost, accuracy)\n",
    "            \n",
    "    # Save Model\n",
    "    saver = tf.train.Saver()\n",
    "    save_path = saver.save(sess, save_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 检查点\n",
    "\n",
    "模型已保存到本地。\n",
    "\n",
    "## 测试模型\n",
    "\n",
    "利用测试数据集测试你的模型。这将是最终的准确率。你的准确率应该高于 50%。如果没达到，请继续调整模型结构和参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import tensorflow as tf\n",
    "import pickle\n",
    "import helper\n",
    "import random\n",
    "\n",
    "# Set batch size if not already set\n",
    "try:\n",
    "    if batch_size:\n",
    "        pass\n",
    "except NameError:\n",
    "    batch_size = 64\n",
    "\n",
    "save_model_path = './image_classification'\n",
    "n_samples = 4\n",
    "top_n_predictions = 3\n",
    "\n",
    "def test_model():\n",
    "    \"\"\"\n",
    "    Test the saved model against the test dataset\n",
    "    \"\"\"\n",
    "\n",
    "    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))\n",
    "    loaded_graph = tf.Graph()\n",
    "\n",
    "    with tf.Session(graph=loaded_graph) as sess:\n",
    "        # Load model\n",
    "        loader = tf.train.import_meta_graph(save_model_path + '.meta')\n",
    "        loader.restore(sess, save_model_path)\n",
    "\n",
    "        # Get Tensors from loaded model\n",
    "        loaded_x = loaded_graph.get_tensor_by_name('x:0')\n",
    "        loaded_y = loaded_graph.get_tensor_by_name('y:0')\n",
    "        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')\n",
    "        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')\n",
    "        \n",
    "        # Get accuracy in batches for memory limitations\n",
    "        test_batch_acc_total = 0\n",
    "        test_batch_count = 0\n",
    "        \n",
    "        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):\n",
    "            test_batch_acc_total += sess.run(\n",
    "                loaded_acc,\n",
    "                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})\n",
    "            test_batch_count += 1\n",
    "\n",
    "        print('Testing Accuracy: {}\\n'.format(test_batch_acc_total/test_batch_count))\n",
    "\n",
    "        # Print Random Samples\n",
    "        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))\n",
    "        random_test_predictions = sess.run(\n",
    "            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),\n",
    "            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})\n",
    "        helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)\n",
    "\n",
    "\n",
    "test_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 为何准确率只有50-80%？\n",
    "\n",
    "你可能想问，为何准确率不能更高了？首先，对于简单的 CNN 网络来说，50% 已经不低了。纯粹猜测的准确率为10%。但是，你可能注意到有人的准确率[远远超过 80%](http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html#43494641522d3130)。这是因为我们还没有介绍所有的神经网络知识。我们还需要掌握一些其他技巧。\n",
    "\n",
    "## 提交项目\n",
    "\n",
    "提交项目时，确保先运行所有单元，然后再保存记事本。将 notebook 文件另存为“dlnd_image_classification.ipynb”，再在目录 \"File\" -> \"Download as\" 另存为 HTML 格式。请在提交的项目中包含 “helper.py” 和 “problem_unittests.py” 文件。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
